Many nonnormalities, one simulation: Do different data generation algorithms affect study results?

Monte Carlo simulation studies are among the primary scientific outputs contributed by methodologists, guiding application of various statistical tools in practice. Although methodological researchers routinely extend simulation study findings through follow-up work, few studies are ever replicated. Simulation studies are susceptible to factors that can contribute to replicability failures, however. This paper sought to conduct a meta-scientific study by replicating one highly cited simulation study (Curran et al., Psychological Methods, 1, 16-29, 1996) that investigated the robustness of normal theory maximum likelihood (ML)-based chi-square fit statistics under multivariate nonnormality. We further examined the generalizability of the original study findings across different nonnormal data generation algorithms. Our replication results were generally consistent with original findings, but we discerned several differences. Our generalizability results were more mixed. Only two results observed under the original data generation algorithm held completely across other algorithms examined. One of the most striking findings we observed was that results associated with the independent generator (IG) data generation algorithm vastly differed from other procedures examined and suggested that ML was robust to nonnormality for the particular factor model used in the simulation. Findings point to the reality that extant methodological recommendations may not be universally valid in contexts where multiple data generation algorithms exist for a given data characteristic. We recommend that researchers consider multiple approaches to generating a specific data or model characteristic (when more than one is available) to optimize the generalizability of simulation results.

Evaluating Close Fit in Ordinal Factor Analysis Models With Multiply Imputed Data.

Multiple imputation (MI) is one of the recommended techniques for handling missing data in ordinal factor analysis models. However, methods for computing MI-based fit indices under ordinal factor analysis models have yet to be developed. In this short note, we introduced the methods of using the standardized root mean squared residual (SRMR) and the root mean square error of approximation (RMSEA) to assess the fit of ordinal factor analysis models with multiply imputed data. Specifically, we described the procedure for computing the MI-based sample estimates and constructing the confidence intervals. Simulation results showed that the proposed methods could yield sufficiently accurate point and interval estimates for both SRMR and RMSEA, especially in conditions with larger sample sizes, less missing data, more response categories, and higher degrees of misfit. Based on the findings, implications and recommendations were discussed.

Factor Structure of the Teacher Strengths and Difficulties Questionnaire in a Large Community-Based Sample: An Investigation of Alternative Measurement Models.

The Strengths and Difficulties Questionnaire (SDQ) is a screening measure commonly used to assess behavioral and emotional symptoms and strengths among children and adolescents. However, despite its frequent use, its underlying factor structure remains an important area of inquiry. Whereas the original five-factor structure has often been supported through exploratory factor analysis, results from confirmatory analyses continue to yield mixed results. We analyzed data from youth in Grades K through 12 from a large epidemiologic study in the Southeastern United States. Teacher-report SDQ data were used to test three confirmatory factor models by school level (i.e., elementary [Grades K-5] and secondary [Grades 6-12]): The original five-factor model, a three-factor model, and a bifactor model. Model fit indices and reliability measures supported the original five-factor model as the preferred model when using the teacher-reported SDQ with both elementary and secondary school children. Implications for using the SDQ in applied research and predictive modeling are discussed.

One step at a time: A statistical approach for distinguishing mediators, confounders, and colliders using direction dependence analysis.

In observational data, understanding the causal link when estimating the causal effect of an independent variable (x) on a dependent variable (y) often requires researchers to identify the role of a third variable in the x → y relationship. Mediation, confounding, and colliding are three key third-variable effects that yield different theoretical and methodological implications for drawing causal conclusions. Commonly used covariance-based statistical methods, such as linear regression and structural equation modeling, cannot distinguish these effects in practice, however. In this study, we introduce a statistical approach for distinguishing mediators, confounders, colliders, and potential M-bias structures that uses higher-order moment information from the data. We propose a two-step procedure that uses the Hilbert-Schmidt independence criterion within the direction dependence analysis framework. Results from Monte Carlo simulations show that our proposed approach accurately recovers the true data-generating process of the third variable. We provide an empirical example to demonstrate the application of our proposed approach in psychological research. Finally, we discuss implications and future directions of our work. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Detecting heterogeneity in the causal direction of dependence: A model-based recursive partitioning approach.

Methods of causal discovery and direction of dependence to evaluate causal properties of variable relations have experienced rapid development. The majority of causal discovery methods, however, relies on the assumption of causal effect homogeneity, that is, the identified causal structure is expected to hold for the entire population. Because causal mechanisms can vary across subpopulations, we propose combining methods of model-based recursive partitioning and non-Gaussian causal discovery to identify such subpopulations. The resulting algorithm can discover subpopulations with potentially varying magnitude and causal direction of effects under mild parameter inequality assumptions. Feasibility conditions are described and results from synthetic data experiments are presented suggesting that large effects and large sample sizes are beneficial for detecting causally competing subgroups with acceptable statistical performance. In a real-world data example, the extraction of meaningful subgroups that differ in the causal mechanism underlying the development of numerical cognition is illustrated. Potential extensions and recommendations for best practice applications are discussed.

The NEAT Equating Via Chaining Random Forests in the Context of Small Sample Sizes: A Machine-Learning Method.

The part of responses that is absent in the nonequivalent groups with anchor test (NEAT) design can be managed to a planned missing scenario. In the context of small sample sizes, we present a machine learning (ML)-based imputation technique called chaining random forests (CRF) to perform equating tasks within the NEAT design. Specifically, seven CRF-based imputation equating methods are proposed based on different data augmentation methods. The equating performance of the proposed methods is examined through a simulation study. Five factors are considered: (a) test length (20, 30, 40, 50), (b) sample size per test form (50 versus 100), (c) ratio of common/anchor items (0.2 versus 0.3), and (d) equivalent versus nonequivalent groups taking the two forms (no mean difference versus a mean difference of 0.5), and (e) three different types of anchors (random, easy, and hard), resulting in 96 conditions. In addition, five traditional equating methods, (1) Tucker method; (2) Levine observed score method; (3) equipercentile equating method; (4) circle-arc method; and (5) concurrent calibration based on Rasch model, were also considered, plus seven CRF-based imputation equating methods for a total of 12 methods in this study. The findings suggest that benefiting from the advantages of ML techniques, CRF-based methods that incorporate the equating result of the Tucker method, such as IMP_total_Tucker, IMP_pair_Tucker, and IMP_Tucker_cirlce methods, can yield more robust and trustable estimates for the "missingness" in an equating task and therefore result in more accurate equated scores than other counterparts in short-length tests with small samples.

Is the Area Under Curve Appropriate for Evaluating the Fit of Psychometric Models?

In the literature of modern psychometric modeling, mostly related to item response theory (IRT), the fit of model is evaluated through known indices, such as χ2, M2, and root mean square error of approximation (RMSEA) for absolute assessments as well as Akaike information criterion (AIC), consistent AIC (CAIC), and Bayesian information criterion (BIC) for relative comparisons. Recent developments show a merging trend of psychometric and machine learnings, yet there remains a gap in the model fit evaluation, specifically the use of the area under curve (AUC). This study focuses on the behaviors of AUC in fitting IRT models. Rounds of simulations were conducted to investigate AUC's appropriateness (e.g., power and Type I error rate) under various conditions. The results show that AUC possessed certain advantages under certain conditions such as high-dimensional structure with two-parameter logistic (2PL) and some three-parameter logistic (3PL) models, while disadvantages were also obvious when the true model is unidimensional. It cautions researchers about the dangers of using AUC solely in evaluating psychometric models.

Applying Negative Binomial Distribution in Diagnostic Classification Models for Analyzing Count Data.

Diagnostic classification models (DCMs) have been used to classify examinees into groups based on their possession status of a set of latent traits. In addition to traditional item-based scoring approaches, examinees may be scored based on their completion of a series of small and similar tasks. Those scores are usually considered as count variables. To model count scores, this study proposes a new class of DCMs that uses the negative binomial distribution at its core. We explained the proposed model framework and demonstrated its use through an operational example. Simulation studies were conducted to evaluate the performance of the proposed model and compare it with the Poisson-based DCM.

The Pull-to-Sit Task: Examining Infant Postural Development in Autism Spectrum Disorder.

OBJECTIVE: To evaluate the predictive relationship between early trajectories of postural and head control during a pull-to-sit task and later autism diagnostic and developmental outcomes. STUDY DESIGN: Using a prospective longitudinal design, postural skills of 100 infants at elevated and low familial likelihood of autism spectrum disorder (ASD) were evaluated using a pull-to-sit task monthly from age 1 month to 6 months. At age 24 months, infants were seen for a developmental and diagnostic evaluation completed by examiners masked to participant group. Latent growth curve models were used to compare early trajectories of pull-to-sit performance in infants later diagnosed with ASD and typically developing infants and to predict developmental outcomes. RESULTS: Pull-to-sit trajectories did not differ in infants with an elevated likelihood of ASD or infants with ASD compared with low-likelihood and typically developing infants, but infants with ASD were more likely to exhibit a head lag by age 4 months. In addition, pull-to-sit trajectories were predictive of social and speech skills 2 years later. CONCLUSIONS: These findings highlight the link between very early pull-to-sit skills and later social and language outcomes. Atypical postural development and persistent presence of head lag may be important early indicators of social and language vulnerabilities, including ASD.

Diagnostic Classification Models for a Mixture of Ordered and Non-ordered Response Options in Rating Scales.

When developing ordinal rating scales, we may include potentially unordered response options such as "Neither Agree nor Disagree," "Neutral," "Don't Know," "No Opinion," or "Hard to Say." To handle responses to a mixture of ordered and unordered options, Huggins-Manley et al. (2018) proposed a class of semi-ordered models under the unidimensional item response theory framework. This study extends the concept of semi-ordered models into the area of diagnostic classification models. Specifically, we propose a flexible framework of semi-ordered DCMs that accommodates most earlier DCMs and allows for analyzing the relationship between those potentially unordered responses and the measured traits. Results from an operational study and two simulation studies show that the proposed framework can incorporate both ordered and non-ordered responses into the estimation of the latent traits and thus provide useful information about both the items and the respondents.

A Monte Carlo Study of Confidence Interval Methods for Generalizability Coefficient.

Computing confidence intervals around generalizability coefficients has long been a challenging task in generalizability theory. This is a serious practical problem because generalizability coefficients are often computed from designs where some facets have small sample sizes, and researchers have little guide regarding the trustworthiness of the coefficients. As generalizability theory can be framed to a linear mixed-effect model (LMM), bootstrap and simulation techniques from LMM paradigm can be used to construct the confidence intervals. The purpose of this research is to examine four different LMM-based methods for computing the confidence intervals that have been proposed and to determine their accuracy under six simulated conditions based on the type of test scores (normal, dichotomous, and polytomous data) and data measurement design (p×i×r and p× [i:r]). A bootstrap technique called "parametric methods with spherical random effects" consistently produced more accurate confidence intervals than the three other LMM-based methods. Furthermore, the selected technique was compared with model-based approach to investigate the performance at the levels of variance components via the second simulation study, where the numbers of examines, raters, and items were varied. We conclude with the recommendation generalizability coefficients, the confidence interval should accompany the point estimate.

Examining the validity and measurement invariance across gender and race of the Implicit Thoughts, Emotion, and Behavior Questionnaire.

Implicit theory has been relatively well-studied in the areas of intelligence and personality but remains less investigated in mental health. This article aims to analyze the psychometric properties of the Implicit Thoughts, Emotion, and Behavior Questionnaire (ITEB-Q; Schleider & Weisz, 2016a). We tested its factorial validity, measurement invariance across gender and two racial groups, as well as criterion validity in a large, diverse sample of adolescents. The sample consists of 817 adolescents aged between 14 and 19 and in 9th-12th grade from a high school in a Southern city in the United States. The confirmatory factorial analysis results corroborate the three-factor structures (thoughts, emotion, behavior) of the ITEB-Q. Multigroup analyses established the scalar invariance across gender as well as across Black and White adolescents. Further, latent mean comparison showed that female adolescents reported lower means on the emotion and thought subscales than male counterparts, and Black adolescents reported higher means across three subscales than White adolescents. The ITEB-Q and its subscales demonstrate good internal reliability and concurrent validity with measures of life satisfaction, resilience, emotional regulation, and emotional problems. Overall, our findings suggest that the ITEB-Q produces valid and reliable scores to assess implicit theory of thoughts, emotion, and behavior among adolescents. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Teacher-student relationships and adolescents' school satisfaction: Behavioural engagement as a mechanism of change.

BACKGROUND: Teacher-student relationships have been linked to various aspects of students' school functioning, including social-emotional well-being in school, but the underlying mechanisms need more investigation. AIMS: In this study, we analysed longitudinal data to test if students' classroom behavioural engagement was a potential mechanism of change that explained how teacher-student relationships affect student school satisfaction. SAMPLE: We used an archival dataset with a sample of seventh graders (ages 11-14, Mage  = 12.7 year) in a middle school in the Southeastern United States. METHODS: Adolescents completed self-report surveys across three waves over the course of 18 months. RESULTS: Longitudinal structural equation modelling analyses revealed that teacher-student relationships were positively associated with positive classroom engagement behaviours and school satisfaction, respectively, at each time, and positive classroom behaviours at Time 2 fully mediated the longitudinal association between teacher-student relationships (Time 1) and school satisfaction (Time 3). CONCLUSIONS: Taken together, results suggested that fostering positive teacher-student relationships to increase students' positive classroom behaviours could be an effective pathway to promote students' satisfaction with school. The applications of the results in educators' and psychologists' work, such as consultation and trainings with teachers, are discussed.

Poisson Diagnostic Classification Models: A Framework and an Exploratory Example.

Assessments with a large amount of small, similar, or often repetitive tasks are being used in educational, neurocognitive, and psychological contexts. For example, respondents are asked to recognize numbers or letters from a large pool of those and the number of correct answers is a count variable. In 1960, George Rasch developed the Rasch Poisson counts model (RPCM) to handle that type of assessment. This article extends the RPCM into the world of diagnostic classification models (DCMs) where a Poisson distribution is applied to traditional DCMs. A framework of Poisson DCMs is proposed and demonstrated through an operational dataset. This study aims to be exploratory with recommendations for future research given in the end.

Neonatal neurobehavior in infants with autism spectrum disorder.

AIM: To investigate neurobehavioral maturation for neonates who are later diagnosed with autism spectrum disorder (ASD). METHOD: In a prospective longitudinal design, neonatal neurobehavior was examined monthly in 1- to 3-month-old infants at elevated and low familial likelihood of ASD (n=60). At 2 years, infants were seen for a clinical best-estimate evaluation, resulting in 18 infants with ASD and 36 typically developing infants. Repeated-measures analysis of variance models were conducted to examine the effects of age, diagnostic group, and their interactions. RESULTS: Neurobehavioral maturation of infants diagnosed with ASD was largely comparable to typically developing infants from 1 to 3 months, with the exception of the development of attention. Object-focused attention was significantly attenuated for infants with ASD beginning at 2 to 3 months and was predictive of social-communication skills 2 years later. INTERPRETATION: This is the first study to prospectively examine neonatal neurobehavior of infants at an elevated familial likelihood of ASD who later received a diagnosis. Despite relatively intact neurological and behavioral maturation in the neonatal period, attention to objects emerged as a key early indicator of ASD. This suggests a complex attentional vulnerability within the first 3 months of life that may be associated with cascading sequelae of social-communication challenges and the emergence of ASD.

Evaluating SEM Model Fit with Small Degrees of Freedom.

Research has revealed that the performance of root mean square error of approximation (RMSEA) in assessing structural equation models with small degrees of freedom (df) is suboptimal, often resulting in the rejection of correctly specified or closely fitted models. This study investigates the performance of standardized root mean square residual (SRMR) and comparative fit index (CFI) in small df models with various levels of factor loadings, sample sizes, and model misspecifications. We find that, in comparison with RMSEA, population SRMR and CFI are less susceptible to the effects of df. In small df models, the sample SRMR and CFI could provide more useful information to differentiate models with various levels of misfit. The confidence intervals and p-values of a close fit were generally accurate for all three fit indices. We recommend researchers use caution when interpreting RMSEA for models with small df and to rely more on SRMR and CFI.

Agreement Between Parental Report and Clinician Observation of Infant Developmental Skills.

Understanding the convergence between parent report and clinician observation measures of development is important and became even more critical during the COVID-19 pandemic as clinician contact with families was significantly limited. Previous research points to inconsistencies in the degree of agreement between parents and clinicians and very little research has examined these associations for infants within the first year of life. This study investigated the association between parent report and clinician observation measures of social communication and motor skills in 27 young infants who were assessed at 9 and 12 months of age. Results suggest a strong relation between clinician and parent rated motor skills, but weak to moderate associations between clinician and parent rated communication skills. Infant temperament played a significant role in parent ratings of infant communication. Together, these results provide support for data collection via parent report or clinician observation of infant motor skills, but suggest that multiple measures of infant communication may be helpful to obtain high-quality, perhaps more accurate, assessment social-communication skills. Specifically, multiple parent report measures along with an observation of parent-infant interactions will likely provide a more rich and accurate characterization of infant social-communication abilities.

A Short Note on Optimizing Cost-Generalizability via a Machine-Learning Approach.

The costs of an objective structured clinical examination (OSCE) are of concern to health profession educators globally. As OSCEs are usually designed under generalizability theory (G-theory) framework, this article proposes a machine-learning-based approach to optimize the costs, while maintaining the minimum required generalizability coefficient, a reliability-like index in G-theory. The authors adopted G-theory parameters yielded from an OSCE hosted by a medical school, reproduced the generalizability coefficients to prepare for optimizing manipulations, applied simulated annealing algorithm to calculate the number of facet levels minimizing the associated costs, and conducted the analysis in various conditions via computer simulation. With a given generalizability coefficient, the proposed approach, virtually an instrument of decision-making supports, found the optimal solution for the OSCE such that the associated costs were minimized. The computer simulation results showed how the cost reductions varied with different levels of required generalizability coefficients. Machine learning-based approaches can be used in conjunction with psychometric modeling to help planning assessment tasks more scientifically. The proposed approach is easy to adopt into practice and customize in alignment with specific testing designs. While these results are encouraging, the possible pitfalls such as algorithmic convergences' failure and inadequate cost assumptions should also be avoided.

Temperamental Shyness and Anger/Frustration in Childhood: Normative Development, Individual Differences, and the Impacts of Maternal Intrusiveness and Frontal Electroencephalogram Asymmetry.

This study used latent growth curve modeling to identify normative development and individual differences in the developmental patterns of shyness and anger/frustration across childhood. This study also examined the impacts of maternal intrusiveness and frontal electroencephalogram (EEG) asymmetry at age 4 on the developmental patterns of shyness and anger/frustration. 180 children (92 boys, 88 girls; Mage = 4.07 years at baseline; 75.6% White, 18.3% Black, 6.1% multiracial/other) participated in the study. Normative development included significant linear decreases in shyness and anger/frustration. Individual variation existed in the developmental patterns. Children with left frontal EEG asymmetry showed a faster decreasing pattern of shyness. Children who experienced higher maternal intrusiveness and had left frontal EEG asymmetry showed a slower decreasing pattern of anger/frustration.

Fitting Latent Growth Models with Small Sample Sizes and Non-normal Missing Data.

This study investigates the performance of robust ML estimators when fitting and evaluating small sample latent growth models (LGM) with non-normal missing data. Results showed that the robust ML methods could be used to account for non-normality even when the sample size is very small (e.g., N < 100). Among the robust ML estimators, "MLR" was the optimal choice, as it was found to be robust to both non-normality and missing data while also yielding more accurate standard error estimates and growth parameter coverage. However, the choice "MLMV" produced the most accurate p values for the Chi-square test statistic under conditions studied. Regarding the goodness of fit indices, as sample size decreased, all three fit indices studied (i.e., CFI, RMSEA, and SRMR) exhibited worse fit. When the sample size was very small (e.g., N < 60), the fit indices would imply that a proposed model fit poorly, when this might not be actually the case in the population.